Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Enable Comet shuffle with AQE coalesce partitions #651

Closed
wants to merge 31 commits into from
Closed

chore: Enable Comet shuffle with AQE coalesce partitions #651

wants to merge 31 commits into from

Conversation

viirya
Copy link
Member

@viirya viirya commented Jul 10, 2024

Which issue does this PR close?

Closes #387.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

@viirya viirya changed the title chore: Remove COMET_SHUFFLE_ENFORCE_MODE_ENABLED chore: Enable Comet shuffle with AQE coalesce partitions Jul 10, 2024
@viirya viirya marked this pull request as draft July 10, 2024 20:36
@viirya
Copy link
Member Author

viirya commented Jul 10, 2024

Some Spark tests need to be updated. Marked this as a draft for now.

@viirya viirya force-pushed the remove_shuffle_enforce branch from bce36c5 to 5e0dd3b Compare July 18, 2024 20:30
@viirya
Copy link
Member Author

viirya commented Jul 18, 2024

All Comet and TPCDS, TPCH tests are passed. Next, I will fix Spark SQL tests.

@viirya viirya force-pushed the remove_shuffle_enforce branch from 17914f6 to 6a9b55b Compare July 24, 2024 00:48
This was referenced Jul 24, 2024
Closed
Closed
Closed
@viirya viirya force-pushed the remove_shuffle_enforce branch 2 times, most recently from 4527010 to afe1176 Compare July 26, 2024 03:54
This was referenced Aug 1, 2024
Closed
Closed
@viirya viirya force-pushed the remove_shuffle_enforce branch 4 times, most recently from 769677f to c406e40 Compare August 2, 2024 21:44
@viirya viirya mentioned this pull request Aug 4, 2024
Closed
@viirya viirya force-pushed the remove_shuffle_enforce branch from c406e40 to 5cbf4f4 Compare August 4, 2024 02:56
@codecov-commenter
Copy link

codecov-commenter commented Aug 4, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 33.99%. Comparing base (4fe43ad) to head (3562b39).
Report is 7 commits behind head on main.

Files Patch % Lines
...ain/java/org/apache/comet/parquet/BatchReader.java 0.00% 1 Missing ⚠️
...l/comet/execution/arrow/CometArrowConverters.scala 0.00% 1 Missing ⚠️
...org/apache/comet/CometSparkSessionExtensions.scala 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main     #651      +/-   ##
============================================
+ Coverage     33.94%   33.99%   +0.05%     
+ Complexity      874      858      -16     
============================================
  Files           112      112              
  Lines         42916    42875      -41     
  Branches       9464     9461       -3     
============================================
+ Hits          14567    14576       +9     
+ Misses        25379    25304      -75     
- Partials       2970     2995      +25

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@viirya viirya mentioned this pull request Aug 4, 2024
Closed
@viirya viirya force-pushed the remove_shuffle_enforce branch 4 times, most recently from b247bb8 to b960536 Compare August 11, 2024 03:58
@viirya viirya closed this Aug 11, 2024
@viirya viirya reopened this Aug 11, 2024
@viirya viirya mentioned this pull request Aug 11, 2024
Closed
@viirya viirya force-pushed the remove_shuffle_enforce branch 2 times, most recently from 6cc5438 to b636f58 Compare August 12, 2024 05:48
@viirya viirya force-pushed the remove_shuffle_enforce branch from 55191f7 to 0e01818 Compare August 15, 2024 19:10
@viirya
Copy link
Member Author

viirya commented Aug 15, 2024 •
edited
Loading

For the negative refcount issue reported by Java Arrow, I cannot reproduce it locally or even on internal CI (different platform other than ubuntu).

I'm not sure if it is related to the ubuntu platform the GitHub Actions uses. Because GitHub Actions only supports ubuntu for linux, I cannot try other linux os. Currently I can only try to use older ubuntu (20.04) and see it helps.

@viirya viirya mentioned this pull request Aug 16, 2024
Merged
@viirya
Copy link
Member Author

viirya commented Aug 16, 2024

Close in favor of #834.

@viirya viirya closed this Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enable Comet shuffle with AQE coalesce partitions
2 participants
@viirya @codecov-commenter